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Background / Context: 

Description of prior research and its intellectual context. 

Due to an increasing interest in the use of single-subject experimental designs (SSEDs), 
appropriate techniques are needed to analyze this type of data. Regression analysis (Center, 

Skiba, & Casey, 1985-1986), and ANOVAs (Gentile, Roden, & Klein, 1972) - amongst other 
models - have been proposed as ways to estimate and/or test the treatment effect per case. 
However, (single-case) researchers may not only be interested in the effect of a treatment for a 
specific case, but also in the effect for a broader range of cases. In order to examine the external 
validity, single-case researchers often replicate their single-subject experiment within a study, 
such as in the multiple -baseline design, or across studies. Multiple-baseline designs across 
participants are the most popular designs amongst the SSEDs (Shadish & Sullivan, 2011) 
because they are characterized by a high internal validity (i.e., the introduction of a treatment is 
staggered across time, which allows disentangling changes in outcome scores due to some 
external event from real treatment effects, Barlow & Herson, 1984; Barlow, Nock, & Hersen, 
2009). These designs are characterized by a hierarchical structure since measurement occasions 
are nested within cases. Ignoring this structure can lead to misleading treatment effect estimates 
and their standard errors. Therefore the use of a two-level regression model, which is a simple 
extension of the linear regression model has been proposed (Van den Noortgate & Onghena, 
2003) and validated using simulation studies (Eerron, Bell, Rendina-Gobioff, & Hibbard, 2009; 
Eerron, Earmer, & Owens, 2010). However, as many multiple-baseline design studies’ results are 
published and because some of them have the same research purpose, there is a good reason to 
estimate the treatment effect not only within a study, but also across studies. This need 
motivates use of the three-level model suggested by Van den Noortgate and Onghena (2003a, 
2003b, and 2008) and validated using intensive simulation studies (Owens & Eerron, 2010, 
Moeyaert, Ugille, Eerron, Beretvas, & Van den Noortgate, 2013a, 2013b; Ugille, Moeyaert, 
Beretvas, Eerron, & Van den Noortgate, 2012). Because previous research has only focused on 
the basic two and three-level model, making a lot of assumptions (e.g., linear trends, continuous 
outcomes, independent, identically and normally distributed errors at the different levels), a lot of 
methodological questions and challenges remain. Because of the flexibility of the multilevel 
model, we will extend the basic two- and three-level model which enables analyzing more 
complex and realistic SSED data. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of this paper proposal is to present four studies (Beretvas, Hembry, Van den 
Noortgate, & Eerron, 2013; Bunuan, Hembry & Beretvas, 2013; Moeyaert, Ugille, Eerron, 
Beretvas, & Van den Noortgate, 2013c; Petit-Bois, Back & Eerron, 2013) each studying 
empirically one particular extension of the basic two-level or basic three-level model that can be 
used to synthesize SSED data across cases (i.e., two-level model) and across cases and studies 
(three-level model). Each extension is discussed and examined through intensive simulation 
studies by looking at the bias and precision of the parameter estimates, and the validity of 
statistical inferences. The results indicate whether the extended two- and three-level models are 
recommended to summarize SSEDs and in which conditions. This is a timely issue, since the 
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number of published SSEDs is inereasing rapidly together with the need for adequate teehniques 
to analyze and meta-analyze SSEDs answering the same underlying researeh question. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

Although the basie two-level and three-level model have been validated and extensively 
studied, a lot of methodologieal issues remain. Therefore, in this paper, we would like to present 
and diseuss four extensions of the basie two- and three-level model. Eaeh extension is validated 
through simulation studies and is an original eontribution to the field of multilevel modeling of 
raw SSED data. In a first part, we present two extensions of the basie two-level model. A first 
extension (i.e. study 1) involves the modeling of dependent (or autoeorrelated) level- 1 error 
struetures and in a seeond extension (i.e., study 2), a model that ean be used for non-linear 
trajeetories is investigated and validated. In a seeond part, we diseuss two extensions of the basie 
three-level model (i.e., study 3 and study 4 respeetively). In study 3, we study standardization in 
situations where the within-ease varianee is heterogeneous and in a fourth study we investigate 
the eovarianee matrix speeifieation at the seeond and third level of the multilevel model. 


Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 

Part 1: Two-Level Modeling 

In a first part, we present two extensions of the basie two-level model. The basie two- 
level model is a simple extensions of a regression equation in whieh the observed seores, i, 
within ease j are regressed on a dummy variable, Dij, indieating the treatment phase (Dij equals 
zero if measurement i within ease j belongs to the baseline, one otherwise): 

Yij = Poj + PijDij + eij eij~N(0, (j|) (1) 

Poj indieates the expeeted baseline level and the eoeffieient ean be interpreted as the effeet of 
the treatment on the outeome. At the seeond level, the eoeffieients of Equation 1 are allowed to 
vary over eases: 


Poj — ^00 + ^0; 
Plj — ^10 + 
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Single-ease researehers are espeeially interested in the estimate of 6io, indieating the average 
estimated treatment effeet aeross eases and in the estimate of , referring to the between-ease 
varianee of this estimated treatment effeet. 

Study 1: Modeling dependencies among the within-case errors. In a first extension, 
the within-ease errors, presented by eij in Equation I, were modeled to follow a first-order 
autoregressive proeess. The level- 1 eovarianee strueture, as shown in Equation 3 for four points 
in time, implies that errors that are eloser together in time eovary more strongly than errors that 
are separated further in time, a pattern that is often reasonable for behavioral data (Baek & 
Eerron, 2013). 
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Similar to the basie two-level model, the level- 1 regression eoeffieients are allowed to vary at the 
seeond level. 
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Study 2: Modeling non-linear trajectories. In a second extension, the focus of interest 
is the modeling of non-linear trends and therefore, the regression equation at the first level looks 
as follows: 


Yij - Poj + “ ^ij) 


f - Poj 

1 -I- exp —a(Tij — b)] 


+ ^B,ij 


During the treatment phase, the model follows a logistic curve where /is the upper right 
asymptote, a is the rapidity with which the trajectory accelerates toward the upper right 
asymptote, and b is the location of the steepest slope for the trajectory. The coefficient, Pqj is 
allowed to vary at the second level. 

Part 2: Three-Level Modeling 

In the second part of this study, we discuss two possible extension of the basic three-level 
model in which a time trend is modeled. The model that will be investigated in this study is a 
multilevel extension of the model of Center, et al. (1985-1986). More specifically, the observed 
scores for case j from study k are regressed on a time indicator, T'ijk^ that is centered around the 
first observation of the intervention phase, a dummy variable, for the treatment phase, and 
an interaction term of these variables, TijkDijk' 

'^ij ~ Pojk "f PljkTijk "f Pljk^ijk "f PsjkTijk^ijk T ^ijk (4) 

Pojk indicates the expected baseline level at the start of the treatment phase (when Tijk - 0)> Pljk 
is the linear trend in the baseline scores. The coefficient P 2 jk can then be interpreted as the 
immediate effect of the intervention on the outcome, whereas P^jk gives an indication of the 
effect of the intervention on the trend. These four coefficients are allowed to vary at the second 
and at the third level. 

Study 3: Standardization when heterogeneous within-case variance is observed. 

Previous research validated the standardizing method suggested by Van den Noortgate and 
Onghena (2008) to standardize SSED data in contexts where the within-case variance is 
homogeneous (Moeyaert et al., 2013b). The standardizing method involves dividing the raw 
SSED data by the estimated within-case standard deviation. However, it is more reasonable to 
assume heterogeneous within-case variance and then the question rises which residual within- 
case standard deviation to use to standardize (i.e., pooled within-case standard deviation, within- 
baseline standard deviation, etc.). In this third study, we suggest three standardizing methods and 
present the empirical validation. 

Study 4: covariance specification at the second and the third level. In previous 
research, the covariance at the second and third level was not estimated. However, it is 
reasonable that the coefficients modeled at the second and third levels covary. Therefore, at the 
second and third level, we estimate covariance between each pair of regression coefficients (i.e., 
covariances among the errors within the level). In this study, we also examine the consequences 
of misspecification of the covariance matrix on the estimated average treatment effects across 
cases and across studies. 

Research Design: 

Description of the research design. 

(May not be applicable for Methods submissions) 
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The extensions of the basic multilevel models were each evaluated by means of an extensive 
simulation study. More specifically, we simulated data for MBD studies with starting points of 
the treatment phase staggered across the series. Based on a study of Shadish and Sullivan (201 1), 
a survey of multiple baseline studies by Perron, Farmer, and Owens (2010), and meta-analyses of 
SSED data (including, e.g., Alen, Grietens, & Van den Noortgate, 2009; Denis, Van den 
Noortgate, & Maes, 2011; Kokina & Kem, 2010; Shogren, Fagella-Luby, Bae, & Wehmeyer, 
2004; Swanson & Sachse-Lee, 2000; Wang, Cui, & Parrila, 2011), we varied a number of 
variables. The chosen variables depend on the specific study. For instance, in studies 3 and 4, 
more variables were varied in comparison to studies 1 and 2, such as the number of studies, the 
between-study variance and other study- specific variables. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

Data are simulated and analyzed in SAS, using the restricted maximum likelihood estimation 
procedure implemented in the procedure MIXED for multilevel or mixed models and using 
MCMC estimation - using SAS’s MCMC procedure - for the nonlinear model study. 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

In the first study, treatment effect estimates were found to be unbiased (and the inferences 
accurate) when the level- 1 error structure was assumed to be first-order autoregressive and the 
data were generated based on either a first-order autoregressive model, a moving average model, 
or an independence model. In the second study, the use of a logistic function for nonlinear 
treatment phase trajectories was investigated. This study supports use of the logistic function to 
describe MBD trajectories when the data exhibit upper asymptotes during the intervention phase. 
The study also supports use of the simpler change in level model and of a model with a quadratic 
function when the series are sufficiently long. The third study, in which several methods for 
standardizing data from MBDs across participants and settings are investigated in scenarios with 
heterogeneous phase variances, shows that one standardizing method clearly works better in 
terms of fixed effect parameter recovery. In the final study, possible covariances among the 
level-2 errors, and among the level-3 errors, were considered and again fixed effects were 
unbiased, but under some conditions problems arose in the fixed effect inferences. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

In this proposal, we presented four studies that are part of a larger study on combining the results 
of SSED data (funded by lES Grant R305D1 10024). We conclude that the multilevel model is 
flexible enough to be extended and is found to be appropriate for combining SSEDs. However, 
there are still a lot of other extensions that are not discussed here that are being investigated and 
that still remain to be investigated such as other estimation procedures (i.e., bootstrapping and 
Bayesian estimation), count outcomes, and multiple dependent outcome variables. Some of these 
topics will be shortly mentioned during our presentation. 
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